posted 11-13-2007 04:08 PM            

can anyone with an academic inclination provide me with what we, in the profession, might consider a "normal" false negative rate. I do not mean to include those with CM usage, just regular wrong calls where the person is lying and is considered NDI.

posted 11-13-2007 06:14 PM            

Lab studies:

Average of 8% false negatives (0 to 15% in nine studies).

Field studies:

Average of 12% false negatives (0 to 24% in four studies).

posted 11-14-2007 12:35 PM            

Its an old adage that there are three kinds of lies:

1) Lies,
2) Damn Lies, and
3) Statistics

(probably from Disraeli, but sometimes attributed to Twain)

There are few places when that famous quotation is more accurate than in discussions of this question - FNs and FPs.

The common methods, in polygraph, for calculating hits and errors are Bayesian. Equally common, if not more common, to academics, researchers, diagnosticians, signal detection professionals, and test designers of all types are inferential methods.

Bayesians calculate results using observed hits and errors only. Inferentialists focus their attention on how well a test result fits a defined model (for truthfulness or deception).

Those models are defined by studying known or simulated data, through descriptive parameters of the distribution of the data. Inferential models are sometimes called "parametric" because of our need to know the shape and parameters of the data. For example, we might have data of lognormal distribution shape (skewed to the right), or normal distribution shape (symmetrical), with this-here mean (center), and that-there standard deviation (dispersion - because the data are never completely black or white, but scatter out quite a bit). There are also nonparametric inferential methods (an example would be the well-known Kruskal-Wallis test, an algorithm that is included in OSS-3 to determine whether the RQS differ significantly between themselves).

Bayesian methods don't require the same understanding or assumptions of the data as inferential methods, and so are simpler to put into actual practice rapidly. They are also easier for some to understand, because the calculations are basic math and basic algebra.

The downside of Bayesian methods is that they are what we call "non-robust" or "non-resistant" to base-rates. This "resistance" is an important concept, because it informs us about much our observed results (numbers) are influenced by base-rates (or whatever factor to which our method is resistant or non-resistant.

Inferential methods are resistant to base-rates.

Hint: you don't have to be strictly a Bayesian or inferentialist - but we humans sometimes like to force dichotomies where they are unnecessary.

Raskin and Honts most likely used Bayesian methods - I say that because a lot of the Utah folks are familiar with Bayesian models, and used both inferential methods (discriminate analysis) and Bayesian models in developing the Utah methods and CPS-II.

OSS (versions 1, 2, and 3) uses an inferential decision model, which was first proposed by Barland in 1985 and which would be instantly recognizable to statisticians, and signal detection researchers everywhere.

Of course, Polyscore uses Logits, which are actually fun once you understand them, and a good solution to polygraph problems. It would be fun to try to apply a Logit decision model to more traditional polygraph physiology (Kircher features), but that will probably never happen due to things like patents and intellectual property (not that there's anything wrong with making money for good work - its just that further development is constrained).

Axciton and Bruce White appear to use a ranking system, and signal detection model (last I checked), but there is scant little information of any actual usefulness, that describes how those methods actually work.

I have seen no description of the mathematical foundations of Identifi, and the information/content on the printed report reveals a misapplication or misunderstanding of the concept of reliability.

--------

So, back to the question about base-rates.

Most laboratory, and most field polygraph studies use samples that include deceptive and truthful subjects in equal proportions. Blackwell (1999) is an example of one that did not, and included 65 deceptive subjects and 35 truthful subjects (I would prefer it to have seen it the other was around).

For most samples of 100 subjects, we will have 50 truthful persons and 50 deceptive persons. While equal base-rates makes for a mathematically convenient sample,in which our observed non-resistant bayesian FN and FP rates are unbiased by the sample base-rate. But there is a huge problem – our knowledge and understanding of Bayesian error rates are not easily generalizable to other situations in which base-rates are unknown. It is largely incorrect and irresponsible to attempt to generalize bayesian error rates to situations with unknown base-rates. More importantly, Bayesian error rates are massively biased with extreme base rates.

Here's the rub – traditional, within group, error rates (as opposed to bayesian, cross-group error rates) are both resistant and generalizable. We should be using these.

I'll put up some data in a bit.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-14-2007 01:47 PM            

I have some data from a recent monte carlo experiment. I'm not satisfied with Zelicoff's solution.

On the other hand, I believe the within-group approach to error calculation does generalize to unknown base rate situations.

I have data and results from a monte carlo experiment to show it. Monte-carlo simulations are really cool, because they allow use to study data, questions, and numbers that would be darn near impossible to get to by other means (like the variance of truthful and deceptive spots in mixed issues exams, for example). I'll post it soon, but I have a conference call waiting.

We ought to try to write a short piece on this.

I don't mind making sense out of the grunt/trench stuff, but I'm not a good writer.

Could you help?

posted 11-14-2007 03:13 PM            

Ray, if you or Barry write something up on this, PLEASE put it in English for the rest of us.

Most of us practitioners are simple minded folk. Anything including the "kibbles and bits" theorums arena will simple cause us to look in the mirror, shrug our shoulders then go to the fridge and get another beer....

Thanks,

Jim

posted 11-14-2007 06:29 PM            

Its gonna get a little worse for a bit.

But after that we can pass out some crayolas and we can shake it off by coloring on some poor SOB's pic. Should we take nominees, or draw names from a hat?

hmm, photoshop and beer - could be dangerous - better wear a helmet.

(note to self: search google for images of Jim Sackett, construction helmets, and beer)

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-14-2007 11:20 PM            

if you can find a pic of me anywhere; I WILL know exactly who gave it to you.

3rd party, and you know who you are, BEWARE!

posted 11-15-2007 01:40 AM            

Start with a basic premise of 100 exams (50 truthful, and 50 deceptive) with an accuracy of .86 (for both sensitivity and specificity).

N=100
base rate=0.5
truthful=50
deceptive=50
accuracy=0.86
TN=43=0.86
FP=7=0.14
TP=43=0.86
FN=7=0.14
FPI=0.14
NPV=0.86

Now look at the same model when we factor in inconclusive observances at the rate of 15%. It does'nt really change much.

N=100
base rate=0.5
truthful=50
deceptive=50
INC=0.15
INC truthful=8
INC deceptive=8
accuracy=0.86
TN=36 (0.72)
FP=6 (0.12)
TP=36 (0.72)
FN=6 (0.12)
FPI=0.143
NPV=0.857

Now look at the results when we set the base rate at .1. The actual proportions of errors don't look much different. However, Bayesian errors are calculated using frequencies, not proportions, as the False Positive Index (FPI = FP / FP+TP)

N=100
base rate=0.1
truthful=90
deceptive=10
INC=0.15
INC truthful=14
INC deceptive=2
accuracy=0.86
TN=65 (0.72)
FP=11 (0.12)
TP=7 (0.70)
FN=1 (0.10)
FPI=0.611
NPV=0.985

A look at the data above also reveals that the Negative predictive value (NPV = proportion of truthful results that are accurate) goes up, under the low base-rate condition, while the FPI goes up considerably (nonresistance). Meaning that when someone passes a properly constructed test, there is little reason to be concerned about an error.

Now look at a very low base-rate example. For this example, we use N=1000, because we can illustrate the precision more clearly with the larger numbers.

N=1000
base rate=0.01
truthful=990
deceptive=10
INC=0.15
INC truthful=149
INC deceptive=2
accuracy=0.86
TN=723 (0.73)
FP=118 (0.12)
TP=7 (0.70)
FN=1 (0.10)
FPI=0.944
NPV=0.999

So which is it? Is the polygraph .99 percent accurate for detecting truthfulness, or .94 inaccurate at detecting deception? This is neither a fair or mathematically appropriate question, but its one which critics like to level at us, just to watch us stutter and fumble. Its also not appropriate to average these two to something like 50 percent – which is what Lars Madson did in front of a room full of people at ATSA in 2003. I'm still kicking myself for remaining too well-manner to confront the error and pick a fight in public at the time. Weighted averaging would have been much more appropriate. Recognizing the limitations imposed by Bayesian nonresistance would be better. Now I think Lars is a co-author of a book, with Dr. Grubin (argh!)

Also, keep in mind that this is an oversimplified illustration, and it makes a lot of uniform assumptions that are not representative of field polygraph situations.

OK, 'nuff-o-that. Next up we'll look at the results of a monte carlo experiment, using a couple of million random numbers to simulate 3-position scoring under various base rates, to see how resistant within-group error prediction models can be against base rate changes.

Ya'll sleep well.

zzzzzzz

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-15-2007 07:54 AM            

I found some pics of Sackett at this link;
www.Sackett/JimPartying@PornConventionW/dwarfMudWrestlerHermaprodites WITH9Handcuffs//whipcream.com

I believe Jim is the tall one with sun glasses.

posted 11-15-2007 11:09 AM            

don't we have policies against spam and trolls???
-------

Mornin' folks,

This 'll be the end of this topic for me, 'cause its requiring too much effort, and I'll get behind on work.

Here are the results of a monte carlo experiment using various base rate levels to illustrate the generalizability of within-group error estimations to varying and unknown base rate situations.

Each of the three monte carlos is a simulation of 3 positions scoring of 3 question ZCT exams (single issue) built from the averages of 10 trials, each of which consists of 300 x 100 x 100 simulated examinations (which is a lot smaller number than 1000 or 10,000 in a more protracted experiment, but its a heckofalot of random numbers) for each of the three base-rate conditions.

Decisions were made using Alpha at .05, which included a Bonferonni correction to .0125 for DI classification, to reduce the well-known mathematical phenomena of increased FPs (due to the addition rule for dependent probability events) when calculating multiple simultaneous significance tests. Alpha at .05 corresponds roughly to hand scores of +2 and -5 in this scoring situation.

For fun, random measurement noise was added to 15% of the data.

Using a base rate of .5 shows:

BR=.50
%Correct= .930 (95%CI=0.925 to 0.936)
INC= .115 (95%CI=0.126 to 0.104)
Sensitivity= .750 (95%CI=0.731 to 0.769)
Specificity= .897 (95%CI=0.887 to 0.908)
FN= .079 (95%CI=0.071 to 0.088)
FP= .044 (95%CI=0.038 to 0.050)

Next we set base rate to a nice low .1 and get this:

BR=.1
%Correct= .951 (95%CI=0.947 to 0.956) (t = -0.81, p=.21)
INC= .069 (95%CI=0.061 to 0.077) (t = 0.96, p=.83)
Sensitivity= .746 (95%CI=0.713 to 0.778) (t = -0.01, p=.50)
Specificity= .901 (95%CI=0.893 to 0.910) (t = -0.10, p=.50)
FN= .078 (95%CI=0.060 to 0.095) (t = 0.13, p=.49)
FP= .042 (95%CI=0.038 to 0.046) (t = -0.01, p=.50)

Then we reduce the base rate further, to a real needle-in-the-haystack level of .01 and see:

BR=.01
%Correct= .956 (95%CI=0.951 to 0.960) (t = -1.08, p=.14)
INC= .060 (95%CI=0.053 to 0.068) (t = 1.27, p=.90)
Sensitivity= .751 (95%CI=0.678 to 0.823) (t = -0.06, p=.48)
Specificity= .900 (95%CI=0.892 to 0.910) (t = 0.23, p=.58)
FN= .072 (95%CI=0.029 to 0.116) (t = -0.02, p=.49)
FP= .041 (95%CI=0.037 to 0.045) (t = -0.06, p=.53)

and finally, an experiment with needle/haystack level set at 1:1000

BR=.001
%Correct= .930 (95%CI=0.925 to 0.936) (t = 1.03, p=.84)
INC= .116 (95%CI=0.105 to 0.127) (t = -1.22, p=.11)
Sensitivity= .744 (95%CI=0.725 to 0.763) (t = -0.03, p=.48)
Specificity= .901 (95%CI=0.890 to 0.911) (t = 0.01, p=.50)
FN= .082 (95%CI=0.074 to 0.910) (t = 0.15, p=.56)
FP= .041 (95%CI=0.035 to 0.046) (t = -0.01, p=.50)

You'll see the inc rate shift around a bit in the above data, that is due most likely to the noise added to the data, to the small number of trials (10, when we would prefer to use 100 or 1000, or better yet 10,000 trials - but that becomes an unweildy number of numbers for a small discussion like this, and to the poor quality random numbers from the M$ Excel97 number generator. I normally use a better, open source, number generator.

Mostly what you'll notice is that within-group error estimations , calculated by subtracting inconclusives from the inverse of the sensitivity and specificity estimates (which are correctly calculated with inconclusives), are resistant to base rate differences.

OK, 'nuff–o–this. I'm off to work.

r(aynman)

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-15-2007 11:48 PM            

quote:

---

Raymond:
Ten minutes to Wapner. We're definitely locked in this box with no TV.

---

posted 11-16-2007 10:03 AM            

Get 'em while they last. They make great holiday gifts.

posted 11-16-2007 11:50 AM            

Skip, NO FAIR! Former supervisors are exempt from participation in the photoshop match game...please!!?? What's left of my reputation is in your hands...:-)

Jim

P.S. I can't believe you guys let my posting stand with that "bad word"...guess it wasn't all that bad.

posted 11-16-2007 03:56 PM            

If you can make me understand it, then I can make it understandable to others.

posted 11-16-2007 05:52 PM            

Maybe you could do something about his looks? Don't you have Photoshop?
----------
All of us really do need a basic understanding of research methodology and statistics. I can follow along rather well - not that it didn't take some effort to study - but I don't trust myself (yet) with the math.

People are willing to sell us a boat load of trash, but tell us it's gold. Discerning between the two isn't all that difficult with a little knowledge, but without it, that trash can look really good.

Ray is clearly pushing us well beyond our comfort zones, but look what has been accomplished. He can't be a one-man band forever. We all need to be able to defend what we do.

You'll notice on the anti cite, not even the guys with a stats background could argue with Ray. They either ignored him or, in a demonstration of their brilliance, mocked him. I used some real simple math to show "Sarge" that polygraph actually helps a majority of those for whom he is concerned, but he didn't get it - or didn't want to get it.

We'll never convince those that are so emotional about the issue that they can't see straight, but, the truly open and curious can be persuaded. I don't know about you, but when I see one side of a debate turn to yelling, screaming, and name-calling, I don't need to know anything about the issue to know whose foundation just crumbled.

If nothing else, get Lou Rovner's "little" research articles in the APA magazine (easy to follow, and easy to share, but big on good info). Then you can jump right into Ray's computations. (Just don't look down before the jump!)

Really, following the math isn't that tough. It's understanding all the nuances well enough to know what math to apply in a given situation that makes me scratch my head.

posted 11-16-2007 09:02 PM            

I'll be here all week.

Try the veal.

(I'm usually not accused of being funny, but some kind of too-serious SOB).

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-16-2007 11:36 PM         

I think the photo-shop shame calendar will be a hot item. I was also thinking about marketing Infidelity Polygraph Gift Certificates-the perfect gift for a loved one at Christmas! What do you think?

posted 11-17-2007 09:07 AM            

Hey, think about this. If you really want to maximize your objectives of detection, disclosure, and deterrence of infidelity, you could perhaps increase your sales revenue and service to customers with a three-pack discount of those gift certificates.

How's this for an add slogan:

"Like buyin' new shorts - one at a time just doesn't do it."

whatdayathink?

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-17-2007 09:53 AM         

Ya Baby...we is on a roll now! How about a spot on the Home Shopping Network?